Benchmarking seven different linear classifier models to determine the best one for Predictoor & trader profits Summary
This benchmarking blog post tries to answer the question, “Which linear classifier model makes the most $”? So we benchmarked all seven linear classifier Predictoor models over 180 days (50k 5min candle iterations), to show absolute value profitability. This time frame is 10x longer than that of previous blog post benchmarks, which were only 18 days (5k iterations). Therefore, a 180-day time frame better shows an absolute value profit which helps determine the best models for Predictoor & trader bots.
Predictoor Profit vs Time for the most successful model, ClassifLinearRidge with None calibration
Over the 180-day term, Predictoor profit fluctuates where some 18-day periods make $, lose $, or remain relatively flat. This is demonstrated in the plot above of Predictoor Profit vs Time for the ClassifLinearRidge model with None calibration, the most successful model for Predictoor profit.
That’s why it’s important to benchmark over longer time frames (180 days+) rather than just short time frames if we want to understand absolute value profitability. Nonetheless, short 18-day benchmarks are useful to compare relative performance of one model vs another (but one cannot draw definitive conclusions about profitability).
Trader Profit vs Time for ClassifLinearRidge with None calibration
A plot of Trader Profit vs Time for the same model also shows how 18-day periods within a 180-day time frame, will either make $, lose $, or remain flat.
This blog post benchmarks Ocean Predictoor simulations for all the Predictoor linear classifier models: ClassifLinearLasso, ClassifLinearLasso_Balanced, ClassifLinearRidge, ClassifLinearRidge_Balanced, ClassifLinearElasticNet, ClassifLinearElasticNet_Balanced, and ClassifLinearSVM. Each implementation is compared with three different calibrations.
This blog post then proceeds to do a walk-through of each of the benchmark plots for predictoor/trader profit, and comparisons of the models & their calibrations.
1. Introduction 1.1 What is Ocean Predictoor?
For information about Ocean Predictoor, please refer to the Predictoor Series blog post that catalogs all the blog posts, articles, and talks related to Predictoor. Learn about ML concepts such as classification, L1 & L2 regularization, calibration, and Predictoor’s simulation tool (“pdr sim”) and (“pdr multisim”) in the Regularized Linear Classifiers With Calibration blog post. Learn about ML balancing in the blog post, The Effects of Balancing on Calibrated Linear Classifiers.
1.2 Benchmarks Outline
We run benchmarks on the approaches:
ClassifLinearLasso & ClassifLinearLasso_Balanced — L1 Regularization. ClassifLinearRidge & ClassifLinearRidge_Balanced— L2 Regularization. ClassifLinearElasticNet & ClassifLinearElasticNet_Balanced — L1 & L2 Regularization. ClassifLinearSVM— L2 Regularization.
The models are benchmarked with the same three calibration approaches, None, Isotonic, and Sigmoid, as in the Linear SVM Classifier with Calibration blog post.
1.3 Experimental Setup
The models were trained on BTC-USDT & ETH-USDT data from Jan 1, 2024 to July 15, 2024. All other experimental parameters, defined in the my_ppss.yaml file, are the same as in the previous blog post, The Effects of Balancing on Calibrated Linear Classifiers.
2. 180-Day Profits of ClassifLinearLasso Balanced & Unbalanced
The ClassifLinearLasso & ClassifLinearLasso_Balanced models are implemented with scikit-learn’s LogisticRegression() function with parameters for a linear kernel trick & L1 penalty.
2.1 ClassifLinearLasso (Unbalanced) 2.1.1 Predictoor Profit
Max Predictoor Profit: 26,545.88 OCEAN
Calibration: None Max_n_train: 2000 Autoregressive_n: 2 2.1.2 Trader Profit
Max Trader Profit: $432.36 USD
Calibration: Isotonic Max_n_train: 1000 Autoregressive_n: 2 2.1.3 Analysis
The ClassifLinearLasso model made moderate Predictoor and trader profits in the 180 days. However, the model maximized Predictoor profit better than trader profit — it generated the third best Predictoor profit of all the benchmarks.
2.2 ClassifLinearLasso_Balanced 2.2.1 Predictoor Profit
Max Predictoor Profit: 6,915.77 OCEAN
Calibration: Sigmoid Max_n_train: 1000 Autoregressive_n: 2 2.2.2 Trader Profit
Max Trader Profit: $639.00 USD
Calibration: Isotonic Max_n_train: 1000 Autoregressive_n: 1 2.2.3 Analysis
While not excelling in Predictoor profits, ClassifLinearLasso_Balanced shows that balancing can improve trader profit returns. This model could be useful where stability and moderate trader returns are desired.
3. 180-Day Profits of ClassifLinearRidge Balanced & Unbalanced
The ClassifLinearRidge & ClassifLinearRidge_Balanced models are implemented with scikit-learn’s LogisticRegression() function with parameters for a linear kernel trick & L2 penalty.
3.1 ClassifLinearRidge (Unbalanced) 3.1.1 Predictoor Profit
Max Predictoor Profit: 41,790.77 OCEAN
Calibration: None Max_n_train: 2000 Autoregressive_n: 2 3.1.2 Trader Profit
Max Trader Profit: $619.43 USD
Calibration: Isotonic Max_n_train: 1000 Autoregressive_n: 1 3.1.3 Analysis
The ClassifLinearRidge model produced the highest Predictoor profit of all the linear classifier model benchmarks. Therefore, it is a good candidate for running with a Predictoor bot over 180 days. It also generated moderate trader profits with Isotonic calibration.
3.2 ClassifLinearRidge_Balanced 3.2.1 Predictoor Profit
Max Predictoor Profit: 12,811.63 OCEAN
Calibration: Sigmoid Max_n_train: 5000 Autoregressive_n: 2 3.2.2 Trader Profit
Max Trader Profit: $897.49 USD
Calibration: None Max_n_train: 2000 Autoregressive_n: 2 3.2.3 Analysis
The ClassifLinearRidge_Balanced model demonstrates strong trader profitability as balancing appears to have boosted trader profits compared to the ClassifLinearRidge model. Interestingly, the model did not need calibration to achieve its large trader profit, whereas previous benchmarks show Isotonic calibration best maximized trader profits.
4. 180-Day Profits of ClassifLinearElasticNet Balanced & Unbalanced
The ClassifLinearElasticNet & ClassifLinearElasticNet_Balanced models are implemented with scikit-learn’s LogisticRegression() function with parameters for a linear kernel trick, L1 & L2 penalties.
4.1 ClassifLinearElasticNet (Unbalanced) 4.1.1 Predictoor Profit
Max Predictoor Profit: 39,109.21 OCEAN
Calibration: None Max_n_train: 2000 Autoregressive_n: 2 4.1.2 Trader Profit
Max Trader Profit: $551.87 USD
Calibration: Isotonic Max_n_train: 1000 Autoregressive_n: 1 4.1.3 Analysis
The ClassifLinearElasticNet model generated the second highest Predictoor profit of the benchmarked models, second only to the ClassifLinearRidge model. Thus, L2 regularization appears to have made both models more accurate than the rest.
4.2 ClassifLinearElasticNet_Balanced 4.2.1 Predictoor Profit
Max Predictoor Profit: 12,709.23 OCEAN
Calibration: Sigmoid Max_n_train: 2000 Autoregressive_n: 2 4.2.2 Trader Profit
Max Trader Profit: $1,172.21 USD
Calibration: None Max_n_train: 2000 Autoregressive_n: 1 4.2.3 Analysis
The ClassifLinearElasticNet_Balanced model achieved the highest trader profit among all benchmarks. As in the other benchmarks, balancing appears to have boosted trader profits. None calibration produced the best trader profit, suggesting that adding classifier calibration to balancing may cause overfitting.
5. 180-Day Profits of ClassifLinearSVM
The ClassifLinearSVM model is implemented with scikit-learn’s LinearSVC() function with parameters for a linear kernel trick and regularization C value of 0.025 (the strength of the regularization is inversely proportional to C).
5.1 Predictoor Profit
Max Predictoor Profit: -162,610.90 OCEAN
Calibration: Sigmoid Max_n_train: 1000 Autoregressive_n: 2 5.2 Trader Profit
Max Trader Profit: $520.30 USD
Calibration: Isotonic Max_n_train: 1000 Autoregressive_n: 1 5.3 Analysis
The ClassifLinearSVM model generated significant losses in Predictoor profit, so it is not recommended for use with a Predictoor bot for 180 days. However, it is possible that tuning the model’s regularization parameter could improve profitability. The model managed to generate moderate trader returns with Isotonic calibration.
6. Analysis and Summary
Which linear classifier model makes the most $?
6.1 Predictoor Profit Analysis
The best Predictoor profit was gained by ClassifLinearRidge model. It gained 41,790.77 OCEAN over the 180 day term with None calibration, max_n_train = 2000, and autoregressive_n = 2. The next best model for Predictoor profitability was ClassifLinearElasticNet. Benchmarks for ClassifLinearSVM model were very poor, losing more than 162k OCEAN during the 180 days. Thus, it should not be used for a Predictoor bot for 180 day terms.
6.2 Trader Profit Analysis
The best trader profit was gained by ClassifLinearElasticNet_Balanced model. It profited $1,172.21 USD with None calibration, max_n_train = 2000, and autoregressive_n = 1. The next best model for trader profitability was ClassifLinearRidge_Balanced.
6.3 Benchmark Trends
Benchmarks show that balancing improved trader profits, especially when paired with L2 regularization, but balancing also reduced Predictoor profits. All the L2 regularized logistic regression models performed best both in Predictoor & trader profit.
6.4 Benchmark Summary
Here’s the breakdown of the best absolute value profitabilities for all seven linear classifier Predictoor models.
ClassifLinearLasso
Max Predictoor profit: 26545.88 OCEAN, calibration = None, max_n_train = 2000, autoregressive_n = 2
Max trader profit: $432.36 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 2
ClassifLinearLasso_Balanced
Max Predictoor profit: 6915.77 OCEAN, calibration = Sigmoid, max_n_train = 1000, autoregressive_n = 2
Max trader profit: $639.00 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 1
ClassifLinearRidge
Max Predictoor profit: 41790.77 OCEAN, calibration = None, max_n_train = 2000, autoregressive_n = 2
Max trader profit: $619.43 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 1
ClassifLinearRidge_Balanced
Max Predictoor profit: 12811.63 OCEAN, calibration = Sigmoid, max_n_train = 5000, autoregressive_n = 2
Max trader profit: $897.49 USD, calibration = None, max_n_train = 2000, autoregressive_n = 2
ClassifLinearElasticNet
Max Predictoor profit: 39109.21 OCEAN, calibration = None, max_n_train = 2000, autoregressive_n = 2
Max trader profit: $551.87 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 1
ClassifLinearElasticNet_Balanced
Max Predictoor profit: 12709.23 OCEAN, calibration = Sigmoid, max_n_train = 2000, autoregressive_n = 2
Max trader profit: $1172.21 USD, calibration = None, max_n_train = 2000, autoregressive_n = 1
ClassifLinearSVM
Max Predictoor profit: -162610.90 OCEAN, calibration = Sigmoid, max_n_train = 1000, autoregressive_n = 2
Max trader profit: $520.30 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 1
7. Conclusion
We benchmarked the absolute value profitability of seven Predictoor linear classifier models over 180 days. The best model for maximizing Predictoor profit was ClassifLinearRidge. It gained 41,790.77 OCEAN over the 180 day term with None calibration, max_n_train = 2000, and autoregressive_n = 2 tunings. The next best model for Predictoor profitability was ClassifLinearElasticNet. The benchmarks also found that the ClassifLinearSVM model was highly negative in Predictoor profitability, losing more than 162k OCEAN over the time frame.
The best model for maximizing trader profit was ClassifLinearElasticNet_Balanced. It profited $1,172.21 USD with None calibration, max_n_train = 2000, and autoregressive_n = 1 tunings. The next best model for trader profitability was ClassifLinearRidge_Balanced.
Throughout the benchmarks, balancing appeared to improve trader profits, especially when paired with L2 regularization, but simultaneously reduced Predictoor profits. Given that the top 2 Predictoor profit models were ClassifLinearRidge & ClassifLinearElasticNet and the top 2 trader profit models were ClassifLinearElasticNet_Balanced & ClassifLinearRidge_Balanced, it appears that L2 regularization of the linear logistic regression models helped to generate the best profits.
8. Appendix: Tables 8.1 ClassifLinearLasso
Max Predictoor profit: 26545.88 OCEAN, calibration = None, max_n_train = 2000, autoregressive_n = 2
Max trader profit: $432.36 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 2
8.2 ClassifLinearLasso_Balanced
Max Predictoor profit: 6915.77 OCEAN, calibration = Sigmoid, max_n_train = 1000, autoregressive_n = 2
Max trader profit: $639.00 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 1
8.3 ClassifLinearRidge
Max Predictoor profit: 41790.77 OCEAN, calibration = None, max_n_train = 2000, autoregressive_n = 2
Max trader profit: $619.43 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 1
8.4 ClassifLinearRidge_Balanced
Max Predictoor profit: 12811.63 OCEAN, calibration = Sigmoid, max_n_train = 5000, autoregressive_n = 2
Max trader profit: $897.49 USD, calibration = None, max_n_train = 2000, autoregressive_n = 2
8.5 ClassifLinearElasticNet
Max Predictoor profit: 39109.21 OCEAN, calibration = None, max_n_train = 2000, autoregressive_n = 2
Max trader profit: $551.87 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 1
8.6 ClassifLinearElasticNet_Balanced
Max Predictoor profit: 12709.23 OCEAN, calibration = Sigmoid, max_n_train = 2000, autoregressive_n = 2
Max trader profit: $1172.21 USD, calibration = None, max_n_train = 2000, autoregressive_n = 1
8.7 ClassifLinearSVM
Max Predictoor profit: -162610.90 OCEAN, calibration = Sigmoid, max_n_train = 1000, autoregressive_n = 2
Max trader profit: $520.30 USD, calibration = Isotonic, max_n_train = 1000, autoregressive_n = 1
Predictoor Benchmarking: 180-Day Profitability of Linear Classifiers was originally published in Ocean Protocol on Medium, where people are continuing the conversation by highlighting and responding to this story.